Crossmodal Prosodic and Gestural Contribution to the Perception of Contrastive Focus

Speech prosody has traditionally been analyzed in terms of acoustic features. Even though visual features and gestures have been shown to help and enhance linguistic processing, the conventional view is that facial and body gesture information in oral (non-sign) languages tends to be redundant and has the role of helping the hearer recover the meaning of an utterance. We conducted two perception experiments with a 3D animated character showing conflicting auditory and visual information to investigate two related questions regarding the importance of gestures in conveying prosodic meaning: (a) how important are facial cues with respect to auditory cues for the perception of contrastive focus?; and (b) what is the relevance of the different gestural movements (i.e., head nod and eyebrow raising) for the perception of this type of focus? Our findings reveal that the visual component is crucial in the semantic interpretation of contrastive focus.